Machine Learning Prague 2025

How well do LLMs perform on text generation tasks, and how can we tell? We present approaches based on annotating individual errors, using human evaluators as well as LLMs. For humans, we introduce our efficient annotation framework and schema. For LLM-based evaluation, we show a metric using an ensemble of open-source LLMs, which includes a reasoning for each annotated error, evaluated on various generation tasks and evaluation aspects (such as accuracy or fluency) and showing high correlation with human annotators. Both approaches allow us to use benchmarks with recent data unseen to LLMs during training, bypassing the data leakage problem that artificially inflates LLMs' performance on commonly used benchmarks.

Ever more decisions are driven by advanced, nonlinear data analysis, where the validity, correctness, and fairness of the outcomes are often assumed but difficult to guarantee in practice. We increasingly rely on the output of algorithmic systems (broader than just LLMs) without fully understanding how they arrive at their results. Although much attention has been paid to the validity and fairness of individual predictions or models, the broader topic of AI engineering and its impact remains relatively unexplored.

AI system design is primarily performed by humans tasked with ensuring that operationalization aligns with business objectives. Currently, alignment is handled opaquely by Data Scientists and/or quantified through performance and fairness metrics.

However, many mistakes occur during design—such as violating causality, linearity, or independence constraints, or introducing bias through seemingly minor engineering choices—due to ignorance or the inability to manage complexity. These issues are typically undetectable by metrics and difficult for humans to identify because of the complex interactions between decisions.

In this talk, we will demonstrate how existing approaches fall short and explain how we believe a human-centric, knowledge-driven AutoML architecture and methodology can make the design process more scientific and systems more trustworthy.

Our goal is to combine the strengths of humans and machines, making the AutoML process explainable and leveraging domain knowledge in the synthesis of pipelines and features to ensure alignment. The architecture explores several novel ideas: first, the construction of pipelines and deep features is approached in a unified way. Next, synthesis is driven by a shared knowledge system, interactively queried to determine which pipeline operations to use or features to compute. Lastly, the synthesis process makes decisions at runtime using partial solutions and the results of their application on data. This approach enables interactive collaboration between humans and machines.

While there has been much excitement about the potential of large language models (LLMs) to automate tasks that previously required human intelligence or creativity, many early projects have failed because of LLMs’ innate willingness to lie. The presentation explores the nature, cause and consequences of this “hallucination” issue and proposes a solution.

By combining generative AI with more traditional symbolic AI, reliability can be maintained, explainability improved and private knowledge and data injected. The talk will show simple examples of combining language-based thinking with computational thinking to generate solutions that neither could achieve on its own.

An example application of an AI scientific research assistant will be shown that brings together the ideas presented in a most demanding real-world task, where false information is not acceptable.

Protein structure prediction has become a cornerstone of modern drug discovery, offering crucial insights for drug-target interactions. This report examines three major computational approaches within an integrated drug development workflow. The primary focus is on AlphaFold 3's architecture, illustrated through simplified implementations of its key components - the Pairformer and Diffusion modules. These demonstration programs provide practical insight into how the system transforms amino acid sequences into accurate three-dimensional structures. RosettaFold is analyzed as a complementary approach, highlighting its comparable performance and distinct advantages in protein design applications. The report also evaluates lightweight protein language models as resource-efficient alternatives for organizations with limited computational infrastructure. Through comparative analysis of these approaches, different prediction strategies can be optimally deployed within the drug development pipeline based on specific requirements for accuracy, speed, and available computational resources.

Breast cancer screening is an indispensable tool in the early detection of one of the most prevalent cancers among women worldwide. However, one persistent challenge lies in accurately diagnosing clusters of microcalcifications (MCs), which are small calcium deposits within the breast tissue. These clusters are notoriously diverse in appearance, making it difficult to distinguish benign from malignant cases with precision. This diagnostic uncertainty often results in unnecessary biopsies, causing undue stress for patients and placing additional burdens on healthcare systems. To address this, we turned to the transformative potential of artificial intelligence (AI), specifically convolutional neural networks (CNNs), to refine the accuracy of breast cancer screening.

Our study investigates two distinct classification approaches using CNNs: a traditional binary method (classifying MCs as either benign or malignant) and a more advanced three-class method (introducing a third category for non-MC cases). Leveraging two robust datasets—the Curated Breast Imaging Subset of the Digital Database for Screening Mammography and the Optimam Database—we identified ResNet-101 as the most effective CNN architecture for this task. To deepen our understanding of the model’s decision-making process, we employed Grad-CAM visualizations, which highlight the regions of mammogram images that most influence the AI’s predictions.

The results revealed a stark contrast between the two approaches. The binary classification model achieved an accuracy of 74.7% and a Matthews correlation coefficient (MCC) of 0.458. While promising, the model displayed notable limitations in interpretability, often relying on surrounding breast tissue rather than focusing directly on the MCs. Additional challenges arose from benign abnormalities, imaging artifacts, breast implants, and excessive black backgrounds in mammogram patches, further complicating accurate predictions.

In contrast, the three-class model brought a leap in performance, achieving an impressive accuracy of 91.7% and an MCC of 0.767. By introducing a third classification category, this model demonstrated an enhanced ability to isolate and evaluate microcalcifications with greater precision. However, this advancement came with a new challenge: vascular calcifications, which were not adequately represented in the training datasets, were occasionally misclassified. These findings underscore the importance of robust, comprehensive datasets to ensure accurate and reliable AI models.

Our research highlights the immense potential of AI to revolutionize breast cancer screening by improving diagnostic accuracy, reducing unnecessary procedures, and enhancing the overall efficiency of healthcare delivery. The significant improvement achieved with the three-class classification approach offers a glimpse into a future where AI can act as a trusted partner for radiologists. However, our findings also emphasize the critical need for continuous refinement of these systems. Addressing the misclassification of vascular calcifications and expanding datasets to encompass a broader range of imaging scenarios will be pivotal in unlocking the full potential of this technology. In this era of rapid technological advancement, our study takes a vital step toward combining cutting-edge AI with human expertise to create a more precise and patient-centered approach to breast cancer detection.

We present an End-to-end AI framework for directed graphs including explainable AI.

This is a machine learning pipeline combining reservoir computing and directed graph analysis to model brain connectivity in stroke patients using MRI data. Effective connectivity is derived via reservoir computing, enabling the creation of directed graph representations. These graphs are classified using a directed graph convolutional networ. Explainable AI tools provide insights into disrupted brain networks, elucidating biomarkers for stroke classification and enhancing clinical interpretability. This approach highlights the potential of machine learning to improve patient stratification in stroke and other brain diseases. The technical innovations are related to reservoir computing networks, directed graph analysis, and explainable AI of effective brain connectivity.

Candy Crush Saga, a popular mobile game with millions of monthly players, leverages AI to ensure gameplay remains engaging, relevant, and fair. The talk consists of two parts. The first part focuses on introducing a novel approach for content recommendation using attentive networks, which was published in RecSys in 2024. We describe the architecture, experimental results, and a scale-adaptive algorithm for fair and relevant recommendations of offers and rewards for completing in-game quests. The second part outlines the implementation of a scalable prediction system for millions of players. We introduce a serving architecture for mobile games, including a typical client-server model. Alongside the talk, we present practical advice on ensuring a robust implementation of recommender systems: detecting degenerate feedback loops, preventing loss of relevance over time, overcoming cold-start problems, and removing bias.

This talk draws from my recent article, which chronicles the evolution of a sports event recommendation system from its early neural network based approach — implemented well before Deep Learning gained mainstream popularity — into a more effective ensemble of simpler scoring methods. These methods range from traditional statistical formulas to sophisticated similarity algorithms leveraging sparse matrix structures.

In this session, we will delve into the technical details of working with sparse matrices, as well as the development and application of the “Hyperbolic Score” evaluation metric, examining why it can be particularly effective under certain conditions. We will also discuss the importance of building a robust Data Platform as a foundation for rapid experimentation, ultimately enabling more efficient development and iteration of recommendation engines.

Beyond the established techniques, I will introduce several thought-provoking ideas for integrating large language models (LLMs) into recommender systems. Additionally, we will explore strategies for balancing model complexity with interpretability, ensuring that both accuracy and transparency remain central concerns in model design.

My hope is that these insights and practical considerations will provide the ML and Data Science community with valuable perspectives on building robust, scalable, and explainable Recommendation Systems.

As much of human activity has moved to the internet in the last few decades, businesses must take into account behavior of individuals and groups online; trends of website or mobile application usage, volume of terms used in search engines, and the popularity pages of specific products and services can all contribute to an understanding of online activity patterns, allowing companies to make informed strategic decisions. Similarweb is one such company to provide these digital insights, termed digital business intelligence, for a myriad of customers and use cases. In the process of generating these metrics, Similarweb combines a small set of actual data with more extensive information from a selected panel of online users. 

A particularly difficult task for Similarweb is the estimation of ad hoc cohorts of users, as clients require the freedom to query and compare their relevant online behavior metric for their specific use case. As it is almost impossible to generate a priori estimations of every possible cohort, it is substantially useful to measure the interaction between each specific user within the cohort and the cohort itself and assign it an individual weight. This weighing problem is similar in essence to problems in recommendation, whose solutions are well established,  but is further complicated by sampling biases within the available actual data, biases within the panel users, the sparsity of interactions between users and cohorts, and the multitude of possible cohorts. We briefly present an earlier approach we have adopted i.e. assigning each panel user a singular weight and applying simple rescaling within a given cohort. This singular weight is useful under strong constraints, but yields poor results on ad hoc cohorts as it cannot account for the non linear nature of interaction between users and cohorts of which they are a part.

We introduce a new multistep approach to create the context dependent weighing. First, we obtain a representative embedding of both common types of cohorts (websites, search terms, product views etc.) and a respective embedding representing users using the same encoder. Second, we train a recommendation-like neural network that learns the non linear interactions between users and their cohorts. This new approach allows us to obtain both an overall sum representation as well as the inner weight distribution for an ad hoc cohort. We demonstrate the usefulness of this new approach on several examples of ad hoc cohorts. It is interesting to note that as an additional byproduct of this training process, we can extract a useful intermediate from the network that embeds both users and cohorts under the constraint of actual data and panel biases.

In a joint research project of the Institute for Intelligent Cyber-Physical Systems ICPS at Heilbronn University and the  industrial partners, SABO Mobile IT GmbH and Grossebacher Systeme AG, research is being conducted into a middleware for voice assistants for global auditory human-technology interaction in the field of industrial plants.
Interactive voice bots are becoming increasingly common in consumer devices and have great potential to support operators and service personnel in operating complex machines. Due to their lack of user-friendliness, flexibility, security and integration into existing industrial solutions, conventional command controls have so far only met with limited acceptance in industrial environments. Chatbots as well as voicebots both involve so-called Natural Language Understanding (NLU) and Natural Language Processing (NLP). However, with voicebots, much more emphasis must be placed on conversation design in order to convey the conversation content in a compact yet understandable way. In addition, a robust method for recognizing voice commands is required. Thanks to their comparatively lower complexity chatbots are already widely used in a variety of applications. Chatbots are mainly used for processes that are easy to automate and always follow a similar pattern. Voicebots, on the other hand, are not or hardly ever used in industrial production environments. The ambient noise level and process reliability are major hurdles. Furthermore, natural communication is not possible with current voicebots. It can be summarized that voicebots are mainly used in acoustically quiet and less critical. 
The aim of the joint research is to realize an assistant for machines that informs the operator, takes over his instructions, checks the exchanged information and thus ensures that a machine works according to the operator's wishes and within its technical capabilities. The deep integration of the intelligent manufacturing assistant bot (IMAB) into the machine software and the understanding of natural language communication is the key to controlling the machine, in contrast to a limited set of commands that humans, not machines, have to learn. The scientific innovation lies in an intelligent language assistant capable of natural language dialogs beyond the level of regular chat bots. A new intermediate layer between a language chat bot-oriented user interface and machine control enables hands-free language communication in combination with classic user interfaces.
The research path will be demonstrated in this talk. Aspects of the resource-related hardware architecture required in an industrial environment and the intelligent software architectures will be presented. In particular, intelligent noise cancellation in the speech signal under a wide range of industrial ambient noise is of crucial importance. Furthermore, the integrated speech-to-text (STT) and text-to-speech (TTS) processes are presented.
In addition, the implementation of an intelligent speaker recognition based on neural networks with third-party speaker detection and an intelligent user authentication based on voice input is demonstrated.
The presentation will cover the latest research results of the process steps within the project requirements. In particular, the scientifically innovative core issues of the research, which are essentially based on machine learning methods, will be demonstrated.

This talk delves into critical aspects of latent fingerprint recognition in crime scene investigations, focusing on fingerprint analysis and system evaluation. We will discuss the challenge of training minutiae detectors without ground truth annotations and introduce innovative approaches using synthetically generated fingerprint data. By leveraging AI and synthetic data, we will show how these advancements can significantly enhance the accuracy and efficiency of fingerprint recognition in forensic science.

Neural networks are ubiquitous yet they remain opaque for most of its users, who has very little understanding of how they store the knowledge and how the information propagates through. In this talk, I would like to share our findings from our quest to understand these phenomena. Specifically I will show the decision rules realized by neural networks and why it might be difficult to understand them without the knowledge of the data distribution. This will give us intuition why neural networks are robust yet why adversarial samples are so easy to create. Finally, we will use these tools to understand, how the decision rules compose during inference.

In this talk, we will explore the field of topic modeling for text documents, focusing on its challenges and practical applications. I will highlight various methods for clustering text documents, enhancing clustering quality, validating results, and integrating solutions into users' daily workflows. Our presentation will share insights and lessons learned from building topic modeling inside a product that serves real customers, emphasizing the challenges of creating scalable systems that deliver real value.

I will emphasize the importance of preprocessing input documents to improve clustering quality. This includes extracting relevant elements from the text, such as entities, key phrases, or summaries. Subsequently, I will demonstrate and compare methods for clustering these text representations, such as hierarchical clustering or directly using LLMs to cluster a large set of documents of text.
In real-world scenarios, new text documents are created daily, and new clusters emerge over time. We will explore techniques to detect new clusters as they appear while maintaining the integrity of existing clusters.

Assessing the quality of discovered topics is a key challenge in topic modeling. I will provide an overview of validation techniques, ranging from traditional machine learning metrics to methods that use LLM as a judge. Furthermore, we will discuss the importance of human-in-the-loop validation processes in ensuring the relevance and accuracy of the topics.

Finally, I will share insights on improving the usability and user experience of topic modeling, including effective naming and description of clusters, and we will discuss whether users are willing to provide feedback on AI models and how this feedback can be used to refine and enhance the existing solutions.

Misinformation poses a growing threat to our society. It has a severe impact on public health by promoting fake cures or vaccine hesitancy, and it is used as a weapon during military conflicts to spread fear and distrust. Current research on natural language processing (NLP) for fact-checking focuses on identifying evidence and predicting the veracity of a claim. People's beliefs, however, often do not depend on the claim and rational reasoning but on credible content that makes the claim seem more reliable, such as scientific publications or visual content that was manipulated or stems from unrelated contexts. To combat misinformation, we need to show (1) "Why was the claim believed to be true?", (2) "Why is the claim false?", (3) "Why is the alternative explanation correct?". In this talk, I will zoom in on two critical aspects of such misinformation supported by credible though misleading content. Firstly, I will present our efforts to dismantle misleading narratives based on fallacious interpretations of scientific publications. Secondly, I will show how we can use multimodal large language models to (1) detect misinformation based on visual content and (2) provide strong alternative explanations for the visual content.

Seznam has been developing custom large language models (LLMs) with a focus on the Czech language and the wide range of Internet services we provide. At the dawn of open-weight models, these early systems lacked several critical capabilities, such as generating fluent and natural Czech text efficiently, understanding nuanced cultural contexts, and managing long-form content. This talk will explore how we addressed these limitations during that formative period and share insights from our journey toward production-ready Czech LLMs.

All major generative AI and Large Language Models (LLMs) have been trained using data scraped from the web.

Additionally, LLM applications often extract web data to provide up-to-date context for answers using Retrieval Augmented Generation (RAG).

However, reliably collecting online data at scale is challenging due to issues like blocking, dynamic content rendering, and the sheer volume of data.

In this talk, we will explain how you can establish an efficient web data extraction pipeline, clean the HTML to circumvent the “garbage in, garbage out” problem, and present examples of successful applications.

As neural networks continue to grow in size and complexity, the demand for efficient models has never been greater. Neural network pruning and quantization have emerged as two of the most promising techniques for reducing model size and improving computational efficiency. But how do these techniques translate from theoretical research to real-world applications?

In this talk, we will explore the state of the art in neural network pruning and quantization, presenting key findings from academia alongside lessons learned from industry implementations.

Using examples from real-world projects, we will discuss practical approaches to algorithm selection, toolchain optimization, and model evaluation. Whether you're a machine learning researcher or a practitioner, this session will equip you with actionable strategies to make neural networks faster, smaller, and more efficient without compromising performance.

Join us to bridge the gap between cutting-edge research and applied machine learning, and discover how to make the most of these transformative techniques in your work.